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(57) Abstract: Noise reduction is 
an important feature in consumer 
television. This is realized by 
spatial, temporal or spatio-temporal 
filters. Spatial filters require pixels 
from within one image, while 
temporal filters require samples 
from two or more successive images. 
The spatio-temporal filter unit 
(100) according to this invention 
integrates spatial and implicit 
motion-compensated temporal noise 
reduction in one filter. For the motion 
compensation no motion vectors 
are required. The spatio-temporal 
filter unit (100) is provided with a 
sigma filter (112) comprising one 
filter kernel (107) designed to operate 
on the pixels from both a current 
image and from the output of the 
spatio-temporal filter unit, being a 
temporally recursive filtered image. 
The operation of the spatio-temporal 
filter unit (100) can be adjusted by 
varying the thresholds of the sigma 
filter (1 12) and the selection of pixels. 
The adjustments can be controlled by 
a motion estimator (222), a motion 
detector (224) and a noise estimator 
(220). 
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Spatio-temporal filter unit and image display apparatus comprising such a spatio-temporal 
filter unit 

The invention relates to a spatio-temporal filter unit, provided with a sigma 
filter, to reduce the noise in images. 

The invention further relates to an image display apparatus comprising such a 
spatio-temporal filter omit 

5 

A unit of the kind described in the opening paragraph is known from G. de 
Haan et al, "Memory integrated noise reduction for television," in IEEE Transactions on 
Consumer Electronics, Vol. 42, No 2, May 1996, Pages 175-181. In this article a unit is 

10 described which is a cascaded system of two filters: a spatial and a temporal one. The spatial 
and temporal filter together should result in effective noise reduction. In general, spatial 
filters require pixels fi-om one image, while temporal filters require samples from two or 
more successive images. 

The spatial filter is a recursive sigma filter. It is based on a variant of the 

15 sigma nearest neighbors selection process. The combination process involves weighted 

averaging, which is optimal for additive white Gaussian noise. The weights of the described 
sigma fillter can have three possible values, which are selected based on thresholds that are 
adapted to the noise level. In the article also a noise estimator is described to estimate the 
noise level. 

20 The temporal filter is realized as a motion compensated first order temporal 

recursive filter. Therefore, this temporal filter requires a motion estimation and motion 
compensation system, which makes the concept expensive. Furthermore, a motion detection 
means is necessary to adapt the strength of the filtering when motion occurs. Otherwise the 
image is distorted. 

25 

It is a first object of the invention to provide a spatio-temporal filter imit, of 
the kind described ui the opening paragraph that integrates spatial and implicit motion- 
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compensated temporal noise reduction in one filter. For the motion compensation no motion 
vectors, or other means to express the amount and direction of movement, are required. 

It is a second object of the invention to provide an image display apparatus 
comprising such a spatio-temporal filter unit. 
5 The Gist object of the invention is achieved in that the spatio-temporal filter 

unit comprises one filter kernel designed to operate on pixels fi:om both a current image and 
firom the output of the spatio-temporal filter unit, being a temporally recursive filtered image. 
The filtering is based on the sigma nearest neighbors selection. Pixels that have poor 
correlation with the original pixel are automatically discarded, whereas other pixels 

10 contribute to the output in proportion to their correlation. It involves weighted averaging of 
selected pixels fi:om a predefined 3-dimensional window comprising horizontal, vertical and 
temporal directions. A major characteristic of this spatio-temporal filter unit is that the spatial 
and temporal filtering are done within the same filter kernel. This reduces the implementation 
cost significantly and leads to an easily optimized concept. Moreover, problems such as the 

15 so-called "dirty- window" eflfect or firozen noise, "comet tails," etc., are automatically solved 
since the temporal filter is no longer separable. Noise breakthrough at edges or around 
moving objects is also eliminated because the spatial and temporal filters complement each 
other. 

An embodiment of the spatio-temporal filter unit according to the invention 

20 comprises: 

- a spatial pixel bujffer to store pixels of the current image; 

- a spatial pixel selector to select pixels firom the spatial pixel buffer; 

- a temporal pixel buffer to buffer pixels fi*om the output of the spatio-temporal filter xmit, 
being a temporally recursive filtered image; 

25 - a temporal pixel selector to select pixels firom the temporal pixel bxiffer; and 

- the sigma filter comprising one filter kernel designed to operate on the pixels firom both the 
spatial pixel selector and the temporal pixel selector. 

In the spatial pixel buffer, a 2-dimensional spatial pixel selector is defined to select pixels to 
enable spatial filtering. In order to add temporally recursive filtered pixels in the filter kernel 
30 of the sigma filter, a second 2-dimensional temporal pixel selector is defined in the temporal 
pixel buffer, which is filled with pixels taken from a picture memory. In this way, 3- 
dimensional spatio-temporal filtering is realized within one filter aperture. The output of the 
sigma filter is stored in a picture memory and used for subsequent filtering. This is the 
temporal recursive part of the processing. Motion detection is inherent in the filtering. Each 
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pixel is first compared with the original one to determine its contribution to the output. 
Where motion occurs, the pixel will have a large difference in grey value. Therefore, it will 
be automatically discarded. Furthermore, motion compensation is also inherent. This is 
reedized by using a temporal pixel selector for the selection of the temporally recursive 
5 filtered pixels and a discriminating averaging filter, which uses a threshold. If there is no 
motion, the grey level correlation relative to the central input pixel is high around the central 
pixel of the previous image. When motion occurs, this correlation is shifted in position 
automatically following the motion trajectory. Provided the motion is low enough, temporal 
filtering will take place because correlated pixels can be found somewhere within the 

10 temporal window, i.e. the temporal pixel selector. In olhcr words, motion compensation is 
implicitly done, thereby making explicit motion compensation redundant. 

An embodiment of the spatio-temporal filter unit according to the mvention 
comprises an adaptive sigma filter. The filter thresholds of the sigma filter are adjustable. 
That means the degree of filtering can be increased or decreased to suit the taste of the user of 

15 this embodiment of the spatio-temporal filter unit or of an image display apparatus 

comprising such a spatio-temporal filter unit. The thresholds for spatial and temporal filtering 
can be independently determined so that the desired balance between spatial and temporal 
filtering is achieved. 

An embodiment of the spatio-temporal filter unit according to the invention is 

20 characterized in that of at least one of the pixel selectors, being the temporal pixel selector 

and the spatial pixel selector, the aperture can be adjusted. This determines the strength of the 
filtering and the characteristics of the output. A combination of a narrow aperture and a large 
aperture with image alternation gives good results. 

An embodiment of the spatio-temporal filter unit according to the invention is 

25 characterized in that at least one of the pixel selectors, being the temporal pixel selector and 
the spatial pixel selector, is designed such that the distance between the selected pixels is 
adjustable. The result is that the aperture of the selector can be adjusted while keeping the 
nimiber of selected pixels more or less constant. The costs of the pixel selectors are based on 
their storage capacity, i.e. the number of pixels they can store simultaneously. The advantage 

30 of this embodiment according to the invention is that the costs for processing of the pixels 
can be relatively low. 

An embodiment of the spatio-temporal filter unit according to flie invention 
comprises a motion detector designed to detect motion and to control the aperture of the 
temporal pixel selector based on detected motion. The aperture determines the strength of the 
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filtering for still images and the range of the motion compensation for moving sequences. An 
advantage of such an embodiment is that the implicit motion compensation is optimized. The 
motion detector may detect global or local motion. As the aperture can be determined on field 
basis, an optimal setting can be foimd for each sequence once the degree of motion activity is 
5 known. The detector does not have to provide information about the direction of movement. 
It is also possible that an external motion estimator provides information about the estimated 
motion. 

An embodiment of the spatio-temporal filter unit according to the invention 
comprises a motion estimator designed to supply motion vectors and to control the position 

10 of the temporal pixel selector related to the temporal pixel buffer, based on the motion 

vectors supplied by the motion estimator. The advantage of such an embodiment is that the 
temporal proximity is increased. This is especially beneficial in the case of high object 
velocities. It is also possible that an external motion estimator provides the motion vectors. 

An embodiment of the spatio-temporal filter unit according to the invention 

1 5 comprises a noise estimator designed to estimate a noise level in the current image and to 

control the aperture of at least one of the pixel selectors, being the temporal pixel selector and 
the spatial pixel selector, based on the estimated noise level. As said the strength of filtering 
is related to the aperture of the pixel selectors. Relatively more filtering is allowed in case of 
a relatively high estimated noise level. Therefore optimal noise filtering is achieved at all 

20 noise levels. It is also possible that an external noise estimator provides the estimated noise 
level. 

An embodiment of the spatio-temporal filter unit according to the invention 
comprises a noise estimator designed to estimate a noise level in the cturrent image and to 
control thresholds of the adaptive sigma filter based on the estimated noise level. The 
25 thresholds can be increased in case of a relatively high estimated noise level and decreased in 
case of a relatively low estimated noise level. Therefore optimal noise filtering is achieved at 
all noise levels. 

30 These and other aspects of the spatio-temporal filter unit and the image display 

apparatus according to the invention will become apparent from and will be elucidated with 
reference with respect to the implementations and embodiments described hereinafter and 
with reference to the accompanying drawings, wherein: 

Fig. 1 schematically shows an embodiment of a spatio-temporal filter unit; 
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Fig. 2 schematically shows an embodiment of a spatio-temporal filter imit 
comprising a noise estimator, a motion estimator and a motion detector; and 
Fig. 3 shows elements of an image display apparatus. 



5 

Figure 1 schematically shows an embodiment of a spatio-temporal filter unit 
100 and two memory imits which are used by the spatio-temporal filter unit 100. These 
memory units might be shared with other units in a system comprising the spatio-temporal 
filter xmit 100. The first memory unit is a picture memory 1 14 to store the ou^ut of the 

10 spatio-temporal filter unit, being a temporally recursive filtered image. The second memory 
imit is a cache 116. The signal representing the images enters the spatio-temporal filter unit 
100 at the connection 102. The filtered signal leaves the spatio-temporal filter unit 100 at the 
connection 118. Pixels fi'om this filtered signal are also stored in the picture memory 1 14. 
Pixels firom an image that enter the spatio-temporal filter unit 100 are buffered in the spatial 

15 pixel buffer 1 04. The spatial pixel selector 1 06 selects pixels firom the spatial pixel bviffer 104 
and provides these to the filter kernel 107 of the sigma filter 112. Pixels firom the picture 
memory 1 14 are cached in the cache 1 16 for reasons of performance. From the cache 116 
pixels are buffered in the temporal pixel buffer 108. The temporal pixel selector 110 selects 
pixels firom the temporal pixel buffer 108 and provides these to the filter kernel 107 of the 

20 sigma filter 1 12. In this way the filter kemel 107 operates on the pixels from both the spatial 
pixel selector 106 and the temporal pixel selector 110. 

Figure 2 schematically shows a relatively more sophisticated embodiment of a 
spatio-temporal filter 200. The white boxes in Figure 2 are part of the spatio-temporal filter 
200. The shaded boxes in Figure 2 are not part of the spatio-temporal filter 200. These shaded 

25 boxes represent memory units. These might be shared with other imits in a system 

comprising the spatio-temporal filter unit 200. The first memory unit is a picture memory 214 
to store the output of the spatio-temporal filter unit, being a temporally recursive filtered 
image. The second memory imit is a cache 216. The signal representing the images enter the 
spatio-temporal filter xmit 200 at the coimection 202. The filtered signal leaves the spatio- 

30 temporal filter xmit 200 at the connection 218. Pixels from this filtered signal are also stored 
in the picture memory 214. Pixels from an image that enters the spatio-temporal filter imit 
200 are buffered in the spatial pixel buffer 204. The spatial pixel selector 206 selects pixels 
from the spatial pixel buffer 204 and provides these to the filter kemel 207 of the sigma filter 
212. Pixels from the picture memory 214 are cached in the cache 216 for reasons of 
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performance. From the cache 216 pixels are buffered in the temporal pixel buffer 208. The 
temporal pixel selector 210 selects pixels from the temporal pixel buffer 208 and provides 
these to the fQter kernel 207 of the sigma filter 2 12. In this way the filter kernel 207 operates 
on the pixels from both the spatial pixel selector 206 and the temporal pixel selector 210. The 
5 spatio-temporal filter 200 comprises a motion detector 224 designed to detect motion and to 
control the aperture of the temporal pixel selector 208 based on detected motion. The spatio- 
temporal filter 200 comprises a motion estimator 222 designed to supply motion vectors and 
to control the position of the temporal pixel selector 210 related to the temporal pixel buffer 
208, based on the motion vectors supplied by the motion estimator. The spatio-temporal filter 
10 200 comprises a noise estimator 220 to estimate a noise level in the current image and to 
control the aperture of at least one of the pixel selectors, being the temporal pixel selector 
210 and the spatial pixel selector 206, based on the estimated noise level. Besides that the 
noise estimator 220 controls thresholds of the adaptive sigma filter 212, based on the 
estimated noise level. 

15 Figure 3 shows elements of an image display apparatus 300 according to the 

invention. The image display apparatus 300 has a receiving means 302 for receiving a video 
signal representing the images to be displayed. The signal may be a broadcast signal received 
via an antenna or cable but may also be a signal from a storage device like a VCR (Video 
Cassette Recorder) or DVD (Digital Versatile Disk). The image display apparatus 300 further 

20 has a spatio-temporal filter unit 304 for filtering the video signal and a display device 306 for 
displaying the images represented by the filtered video signal. The spatio-temporal filter unit 
304 is implemented as described in Figure 1 or Figure 2. 

It should be noted that the above-mentioned embodiments illiistrate rather than 
limit the invention and that those skilled in the art will be able to design alternative 

25 embodiments without departing from the scope of the appended claims. In the claims, any 
reference signs placed between parentheses shall not be constmcted as limiting the claim. 
The word 'comprising' does not exclude the presence of elements or steps other than those 
listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of 
a plurality of such elements. The invention can be implemented by means of hardware 

30 comprising several distinct elements and by means of a suitable programmed computer. In 
the unit claims enumerating several means, several of these means can be embodied by one 
and the same item of hardware. 
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CLAIMS: 



1 . A spatio-temporal filter unit (1 00), provided with a sigma filter (1 12), to 
reduce the noise in images, characterized in that the sigma filter (112) comprises one filter 
kernel (107) designed to operate on pixels from both a current image and from the output 
(1 1 8) of the spatio-temporal filter unit, being a temporally recursive filtered image, 

5 

2. A spatio-temporal filter imit (1 00) as claimed in Claim 1 , characterized in that 
it comprises: 

- a spatial pixel buffer (104) to store pixels of the current image; 

- a spatial pixel selector (106) to select pixels from the spatial pixel buffer (104); 

10 - a temporal pixel buffer (1 08) to buffer pixels from the ou^ut (1 1 8) of the spatio-temporal 
filter unit, being a temporally recursive filtered image; 

- a temporal pixel selector (1 10) to select pixels from the temporal pixel buffer (108); and 

- the sigma filter (112) comprising one filter kemel (107) designed to operate on the pixels 
froni both tiie spatial pixel selector (106) and the temporal pixel selector (1 10). 

15 

3. A spatio-temporal filter unit (100) as claimed in Claim 2, characterized in that 
it comprises an adaptive sigma filter (1 12). 

4. A spatio-temporal filter xmit (100) as claimed in Claim 3, characterized in that 
20 of at least one of the pixel selectors, being the temporal pixel selector (1 10) and the spatial 

pixel selector (106), the aperture can be adjusted. 

5. A spatio-temporal filter unit (100) as claimed in Claim 4, characterized in that 
at least one of the pixel selectors, being the temporal pixel selector (110) and the spatial pixel 

25 selector (106), is designed such that the distance between the selected pixels is adjustable. 

6. A spatio-temporal filter xmit (200) as claimed in Claim 4, characterized in that 
it comprises a motion detector (224) designed to detect motion and to control the aperture of 
the temporal pixel selector (210) based on detected motion. 
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7. A spatio-temporal filter unit (200) as claimed in Claim 4, characterized in that 
it comprises a motion estimator (222) designed to supply motion vectors and to control the 
position of the temporal pixel selector (210) related to the temporal pixel buffer (208), based 

5 on the motion vectors supplied by the motion estimator (222). 

8. A spatio-temporal filtCT unit (200) as claimed in Claim 4, characterized in that 
it comprises a noise estimator (220) designed to estimate a noise level in the cmxent image 
and to control the aperture of at least one of the pixel selectors, being the temporal pixel 

10 selector (210) and the spatial pixel selector (206), based on the estimated noise level. 

9. A spatio-temporal filter unit (200) as claimed in Claim 4, characterized in that 
it comprises a noise estimator (220) designed to estimate a noise level in the current image 
and to control thresholds of the adaptive sigma filter (212), based on the estimated noise 

15 level. 

10. An image display apparatus (300) comprising: 

- receiving means (302) for receiving a signal representing images; 

- a display device (306) for displa3dng the images; and 

20 - a spatio-temporal filter unit (100), provided with a sigma filter (1 12), to reduce die noise in 
the images, characterized in the sigma filter (1 12) comprises one filter kemel (107) designed 
to operate on pixels from both a current image and from the oxitput (1 1 8) of the spatio- 
temporal filter unit, being a temporally recursive filtered image. 

25 1 1 . An image display apparatus (300) comprising: 

- receiving means (302) for receiving a signal representing images; 

- a display device (306) for displaying the images; and 

- a spatio-temporal filter unit (100), provided with a sigma filter (1 12), to reduce the noise in 
the images, characterized in that it comprises: 

30 - a spatial pixel buffer (1 04) to store pixels of a current image; 

- a spatial pixel selector (106) to select pixels from the spatial pixel buffer (104); 

- a temporal pixel bviffer (108) to buffer pixels from the output (1 1 8) of the spatio-temporal 
filter xmit, being a temporally recursive filtered image; 

- a temporal pixel selector (110) to select pixels from the temporal pixel buffer (108); and 
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- the sigma filter (1 12), being an adaptive sigma filter comprising one filter kernel (107) 
designed to operate on the pixels firom both the spatial pixel selector (106) and the temporal 
pixel selector (110). 
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